Conversation assistance audio device personalization

ABSTRACT

Various implementations include conversation assistance audio devices with settings that are adjustable based upon user feedback. In some cases, a computer-implemented method of personalizing a conversation assistance audio device includes: receiving a command from a user for assistance with audio settings in the conversation assistance audio device; applying a baseline set of audio settings to the conversation assistance audio device in response to receiving the user command for assistance; receiving a user command to change the baseline set of audio settings; and applying a modified set of audio settings to the conversation assistance audio device in response to receiving the user command to change the baseline set of audio settings, wherein the modified set of audio settings are applied based upon usage of the conversation assistance audio device and a population model of usage of similar conversation assistance audio devices.

PRIORITY CLAIM

This application is a divisional application of U.S. patent applicationSer. No. 16/165,055, filed on Oct. 19, 2018, which is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure generally relates to audio devices. More particularly,the disclosure relates to conversation assistance audio devices withsettings that are adjustable based upon user feedback.

BACKGROUND

Conversation assistance devices aim to make conversations moreintelligible and easier to understand. These devices aim to reduceunwanted background noise and reverberation. While these devices cansignificantly enhance the day-to-day experience of users with mild tomoderate hearing impairment, these devices can be challenging topersonalize for a particular user.

SUMMARY

All examples and features mentioned below can be combined in anytechnically possible way.

Various implementations include conversation assistance audio deviceswith settings that are adjustable based upon user feedback. In somecases, the conversation assistance audio device is configured to use aset of simulated audio environments to personalize audio settings for auser. In other cases, the conversation assistance audio device isconfigured to iteratively adjust audio settings from a baseline set ofaudio settings in response to user commands.

In some particular aspects, a computer-implemented method ofpersonalizing a conversation assistance audio device includes:presenting a user of the conversation assistance audio device with a setof simulated audio environments played back at the conversationassistance audio device; receiving feedback from the user about eachsimulated audio environment in the set of simulated audio environments;and adjusting at least one audio setting at the conversation assistanceaudio device based upon the feedback from the user and known audiocharacteristics of the set of simulated audio environments and theconversation assistance audio device.

In other particular aspects, a computer-implemented method ofpersonalizing a conversation assistance audio device includes: receivinga command from a user for assistance with audio settings in theconversation assistance audio device; applying a baseline set of audiosettings to the conversation assistance audio device in response toreceiving the user command for assistance; receiving a user command tochange the baseline set of audio settings; and applying a modified setof audio settings to the conversation assistance audio device inresponse to receiving the user command to change the baseline set ofaudio settings, where the modified set of audio settings are appliedbased upon usage of the conversation assistance audio device and apopulation model of usage of similar conversation assistance audiodevice.

Implementations may include one of the following features, or anycombination thereof.

In particular cases, each simulated audio environment in the setincludes audio playback at a substantially identical background noiselevel.

In certain aspects, each simulated audio environment in the set includesplayback of a person speaking at a sound pressure level that is greaterthan the background noise level.

In some implementations, adjusting the at least one audio settingincludes selecting a best-fit audio setting for the conversationassistance audio device based upon the feedback received from the userabout all of the simulated audio environments in the set of simulatedaudio environments.

In particular cases, the set of simulated audio environments includes atleast two simulated audio environments.

In certain implementations, the method further includes: providing aninterface for receiving the feedback from the user, where the interfaceincludes at least one actuatable mechanism for adjusting the at leastone audio setting on the conversation assistance audio device, and wherethe interface is located on the conversation assistance audio device ora computing device connected with the conversation assistance audiodevice.

In some aspects, the at least one actuatable mechanism includes at leastone of:

a weighted scoring mechanism, a volume adjustor or a treble/bassadjustor.

In certain cases, the at least one audio setting on the conversationassistance audio device is adjusted in response to the user modifyingthe at least one actuatable mechanism on the interface.

In particular implementations, the method further includes scaling theadjustment to the at least one audio setting on the conversationassistance audio device based upon a scaling factor.

In some cases, the scaling factor includes an inference between audiocharacteristics of each of the simulated audio environments in the setof simulated audio environments and a model of real-world audiocharacteristics.

In certain aspects, the scaling factor includes a positive scalingfactor, and scaling the adjustment to the at least one audio settingcomprises positively scaling the at least one audio setting on theconversation assistance audio device.

In particular implementations, the method further includes creating aset of distinct audio modes based upon the feedback from the user andknown audio characteristics of the set of simulated audio environmentsand the conversation assistance audio device.

In some cases, the set of distinct audio modes includes a set ofuser-selectable audio modes each correlated with distinct audio filtersconfigured to enhance audibility of a range of frequencies for specificacoustic environments.

In certain aspects, the method further includes switching betweendistinct audio modes in the set of distinct audio modes in response todetecting a change in an acoustic signature of ambient sound detected atthe conversation assistance audio device.

In particular cases, adjusting the at least one audio setting on theconversation assistance audio device includes adjusting a directivity ofa microphone array in the conversation assistance audio device,adjusting a microphone array filter on the microphone array in theconversation assistance audio device, adjusting a volume of audioprovided to the user at the conversation assistance audio device,adjusting parameters controlling wide dynamic range compression oradjusting gain parameters controlling the shape of the frequency versusgain function.

In some aspects, the method further includes providing a suggested audiomode for the conversation assistance audio device to the user based uponthe feedback from the user and the known audio characteristics of theset of simulated audio environments and the conversation assistanceaudio device, prior to adjusting the at least one setting on theconversation assistance audio device.

In certain cases, adjusting the at least one audio setting at theconversation assistance audio device includes applying a signalprocessing modification in at least one of the conversation assistanceaudio device or a computing device connected with the conversationassistance audio device.

In some implementations, each simulated audio environment in the set ofsimulated audio environments comprises audio playback at asignal-to-noise (SNR) range in which audibility limits intelligibility.

In particular aspects, the SNR range is a value between approximately −5and approximately 20.

In certain implementations, the user command for assistance with audiosettings in the conversation assistance audio device includes a singleinterface command.

In some cases, the single interface command includes a single voicecommand, a single tactile command, or a single gesture command.

In particular implementations, the method of personalizing theconversation assistance audio device further includes iterativelyperforming: receiving a subsequent user command to change the modifiedset of audio settings; and applying a further modified set of audiosettings to the conversation assistance audio device in response toreceiving the subsequent user command, where the further modified set ofaudio settings are applied based upon the usage of the conversationassistance audio device and the population model of usage of similarconversation assistance audio device, and where the subsequent usercommand includes a single interface command.

In certain aspects, the baseline set of audio settings includes medianaudio settings for a respective plurality of audio parameters oruser-defined audio settings for the respective plurality of audioparameters from a group of users.

In some cases the audio settings on the conversation assistance audiodevice include at least one of: a directivity of a microphone array inthe conversation assistance audio device, a microphone array filterconfiguration on the microphone array in the conversation assistanceaudio device, a volume of audio provided to the user at the conversationassistance audio device, parameters controlling wide dynamic rangecompression or gain parameters controlling the shape of the frequencyversus gain function.

In particular implementations, the baseline audio settings are appliedin response to the single interface command without any interveningcommand.

Two or more features described in this disclosure, including thosedescribed in this summary section, may be combined to formimplementations not specifically described herein.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features, objectsand advantages will be apparent from the description and drawings, andfrom the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram depicting an example personal audio deviceaccording to various disclosed implementations.

FIG. 2 shows a schematic data flow diagram illustrating controlprocesses performed by a conversation assistance audio engine in thepersonal audio device of FIG. 1.

FIG. 3 shows a process flow diagram illustrating processes performed bythe conversation assistance audio engine shown in FIG. 2.

FIG. 4 shows an example interface for personalizing a conversationassistance audio device according to various implementations.

FIG. 5 shows another example interface for personalizing a conversationassistance audio device according to various implementations.

FIG. 6 shows a set of example interfaces for personalizing aconversation assistance audio device according to variousimplementations.

FIG. 7 shows a process flow diagram illustrating processes performed bythe conversation assistance audio engine shown in FIG. 2.

It is noted that the drawings of the various implementations are notnecessarily to scale. The drawings are intended to depict only typicalaspects of the disclosure, and therefore should not be considered aslimiting the scope of the implementations. In the drawings, likenumbering represents like elements between the drawings.

DETAILED DESCRIPTION

This disclosure is based, at least in part, on the realization thataudio settings in a conversation assistance audio system can bebeneficially tailored to a particular user. For example, a conversationassistance audio system can be configured to present a set of simulatedenvironments to a user in order to elicit feedback for adjusting audiosettings in the audio system. In other cases, the conversationassistance audio system is configured to provide a simplified commandinterface (e.g., a single interface command) for iteratively refiningaudio settings from baseline settings.

Commonly labeled components in the FIGURES are considered to besubstantially equivalent components for the purposes of illustration,and redundant discussion of those components is omitted for clarity.

Conventional hearing assistance devices (or, hearing aids) are typicallytested and adjusted by an audiology professional such as an audiologistin one or more appointments with the user. Interacting with aprofessional on an in-person basis can give the user confidence in thesetup process, and can provide opportunities for refinement of devicesettings as conditions change or evolve.

However, a portion of the population can benefit from devices thatenhance conversation and other select ambient acoustic signals, but maynot wish to use a hearing assistance device and/or seek professionalmedical help. For many of these people, conversation assistance devicesprovide an attractive compromise between a hearing aid and receiving nohearing assistance. Despite the benefits of these conventionalconversation assistance devices, it can be challenging to personalizethe user experience without seeking professional medical help. Examplesof conventional conversation assistance devices are described in U.S.Pat. No. 9,560,451 (“Conversation Assistance System”), which isincorporated by reference here in its entirety.

In contrast to conventional conversation assistance devices, variousimplementations include conversation assistance devices configured for auser with a software module or mobile application that permits the userto personalize the device without consulting an audiologist or otherhearing assistance professional. The approaches described according tovarious implementations present a user with a set of simulated audioenvironments via the conversation assistance audio device. Theapproaches also involve adjusting one or more audio settings on theconversation assistance audio device based upon user feedback and knownaudio characteristics of the set of simulated audio environments and theconversation assistance audio device.

It has become commonplace for those who either listen to electronicallyprovided audio (e.g., audio from an audio source such as a mobile phone,tablet, computer, CD player, radio or MP3 player), those who simply seekto be acoustically isolated from unwanted or possibly harmful sounds ina given environment, and those engaging in two-way communications toemploy personal audio devices to perform these functions. For those whoemploy headphones or headset forms of personal audio devices to listento electronically provided audio, it is commonplace for that audio to beprovided with at least two audio channels (e.g., stereo audio with leftand right channels) to be separately acoustically output with separateearpieces to each ear. For those simply seeking to be acousticallyisolated from unwanted or possibly harmful sounds, it has becomecommonplace for acoustic isolation to be achieved through the use ofactive noise reduction (ANR) techniques based on the acoustic output ofanti-noise sounds in addition to passive noise reduction (PNR)techniques based on sound absorbing and/or reflecting materials.Further, ANR can be combined with other audio functions in headphones,such as conversation enhancing functions, as described in U.S. Pat. No.9,560,451. While the term ANR is used to refer to acoustic output ofanti-noise sounds, this term can also include controllable noisecanceling (CNC), which permits control of the level of anti-noiseoutput, for example, by a user. In some examples, CNC can permit a userto control the volume of audio output regardless of the ambient acousticvolume.

Aspects and implementations disclosed herein may be applicable to a widevariety of personal audio devices, such as wearable audio devices invarious form factors, such as watches, glasses, neck-worn speakers,shoulder-worn speakers, body-worn speakers, etc. Unless specifiedotherwise, the term headphone, as used in this document, includesvarious types of personal audio devices such as around-the-ear,over-the-ear and in-ear headsets, earphones, earbuds, hearing aids, orother wireless-enabled audio devices structured to be positioned near,around or within one or both ears of a user. Unless specified otherwise,the term wearable audio device, as used in this document, includesheadphones and various other types of personal audio devices such asshoulder or body-worn acoustic devices that include one or more acousticdrivers to produce sound without contacting the ears of a user. Itshould be noted that although specific implementations of personal audiodevices primarily serving the purpose of acoustically outputting audioare presented with some degree of detail, such presentations of specificimplementations are intended to facilitate understanding throughprovision of examples, and should not be taken as limiting either thescope of disclosure or the scope of claim coverage.

Aspects and implementations disclosed herein may be applicable topersonal audio devices that either do or do not support two-waycommunications, and either do or do not support active noise reduction(ANR). For personal audio devices that do support either two-waycommunications or ANR, it is intended that what is disclosed and claimedherein is applicable to a personal audio device incorporating one ormore microphones disposed on a portion of the personal audio device thatremains outside an ear when in use (e.g., feedforward microphones), on aportion that is inserted into a portion of an ear when in use (e.g.,feedback microphones), or disposed on both of such portions. Still otherimplementations of personal audio devices to which what is disclosed andwhat is claimed herein is applicable will be apparent to those skilledin the art.

FIG. 1 is a block diagram of an example of a personal audio device 10(e.g., a conversation assistance audio device) having two earpieces 12Aand 12B, each configured to direct sound towards an ear of a user.Features of the personal audio device 10 can be particularly useful as awearable audio device, e.g., a head and/or shoulder-worn conversationassistance device. Reference numbers appended with an “A” or a “B”indicate a correspondence of the identified feature with a particularone of the earpieces 12 (e.g., a left earpiece 12A and a right earpiece12B). Each earpiece 12 includes a casing 14 that defines a cavity 16. Insome examples, one or more internal microphones (inner microphone) 18may be disposed within cavity 16. An ear coupling 20 (e.g., an ear tipor ear cushion) attached to the casing 14 surrounds an opening to thecavity 16. A passage 22 is formed through the ear coupling 20 andcommunicates with the opening to the cavity 16. In some examples, anouter microphone 24 is disposed on the casing in a manner that permitsacoustic coupling to the environment external to the casing.

In implementations that include ANR (which may include CNC), the innermicrophone 18 may be a feedback microphone and the outer microphone 24may be a feedforward microphone. In such implementations, each earphone12 includes an ANR circuit 26 that is in communication with the innerand outer microphones 18 and 24. The ANR circuit 26 receives an innersignal generated by the inner microphone 18 and an outer signalgenerated by the outer microphone 24, and performs an ANR process forthe corresponding earpiece 12. The process includes providing a signalto an electroacoustic transducer (e.g., speaker) 28 disposed in thecavity 16 to generate an anti-noise acoustic signal that reduces orsubstantially prevents sound from one or more acoustic noise sourcesthat are external to the earphone 12 from being heard by the user. Asdescribed herein, in addition to providing an anti-noise acousticsignal, electroacoustic transducer 28 can utilize its sound-radiatingsurface for providing an audio output for playback, e.g., for acontinuous audio feed.

A control circuit 30 is in communication with the inner microphones 18,outer microphones 24, and electroacoustic transducers 28, and receivesthe inner and/or outer microphone signals. In certain examples, thecontrol circuit 30 includes a microcontroller or processor having adigital signal processor (DSP) and the inner signals from the two innermicrophones 18 and/or the outer signals from the two outer microphones24 are converted to digital format by analog to digital converters. Inresponse to the received inner and/or outer microphone signals, thecontrol circuit 30 can take various actions. For example, audio playbackmay be initiated, paused or resumed, a notification to a wearer may beprovided or altered, and a device in communication with the personalaudio device may be controlled. The personal audio device 10 alsoincludes a power source 32. The control circuit 30 and power source 32may be in one or both of the earpieces 12 or may be in a separatehousing in communication with the earpieces 12. The personal audiodevice 10 may also include a network interface 34 to providecommunication between the personal audio device 10 and one or more audiosources and other personal audio devices. The network interface 34 maybe wired (e.g., Ethernet) or wireless (e.g., employ a wirelesscommunication protocol such as IEEE 802.11, Bluetooth, Bluetooth LowEnergy, or other local area network (LAN) or personal area network (PAN)protocols).

Network interface 34 is shown in phantom, as portions of the interface34 may be located remotely from personal audio device 10. The networkinterface 34 can provide for communication between the personal audiodevice 10, audio sources and/or other networked (e.g., wireless) speakerpackages and/or other audio playback devices via one or morecommunications protocols. The network interface 34 may provide either orboth of a wireless interface and a wired interface. The wirelessinterface can allow the personal audio device 10 to communicatewirelessly with other devices in accordance with any communicationprotocol noted herein. In some particular cases, a wired interface canbe used to provide network interface functions via a wired (e.g.,Ethernet) connection.

In some cases, the network interface 34 may also include a network mediaprocessor for supporting, e.g., Apple AirPlay® (a proprietary protocolstack/suite developed by Apple Inc., with headquarters in Cupertino,Calif., that allows wireless streaming of audio, video, and photos,together with related metadata between devices) or other known wirelessstreaming services (e.g., an Internet music service such as: Pandora®, aradio station provided by Pandora Media, Inc. of Oakland, Calif., USA;Spotify®, provided by Spotify USA, Inc., of New York, N.Y., USA); orvTuner®, provided by vTuner.com of New York, N.Y., USA); andnetwork-attached storage (NAS) devices). For example, if a user connectsan AirPlay® enabled device, such as an iPhone or iPad device, to thenetwork, the user can then stream music to the network connected audioplayback devices via Apple AirPlay®. Notably, the audio playback devicecan support audio-streaming via AirPlay® and/or DLNA's UPnP protocols,and all integrated within one device. Other digital audio coming fromnetwork packets may come straight from the network media processorthrough (e.g., through a USB bridge) to the control circuit 30. As notedherein, in some cases, control circuit 30 can include a processor and/ormicrocontroller, which can include decoders, DSP hardware/software, etc.for playing back (rendering) audio content at electroacoustictransducers 28. In some cases, network interface 34 can also includeBluetooth circuitry for Bluetooth applications (e.g., for wirelesscommunication with a Bluetooth enabled audio source such as a smartphoneor tablet). In operation, streamed data can pass from the networkinterface 34 to the control circuit 30, including the processor ormicrocontroller. The control circuit 30 can execute instructions (e.g.,for performing, among other things, digital signal processing, decoding,and equalization functions), including instructions stored in acorresponding memory (which may be internal to control circuit 30 oraccessible via network interface 34 or other network connection (e.g.,cloud-based connection). The control circuit 30 may be implemented as achipset of chips that include separate and multiple analog and digitalprocessors. The control circuit 30 may provide, for example, forcoordination of other components of the personal audio device 10, suchas control of user interfaces (not shown) and applications run by thepersonal audio device 10.

In addition to a processor and/or microcontroller, control circuit 30can also include one or more digital-to-analog (D/A) converters forconverting the digital audio signal to an analog audio signal. Thisaudio hardware can also include one or more amplifiers which provideamplified analog audio signals to the electroacoustic transducer(s) 28,which each include a sound-radiating surface for providing an audiooutput for playback. In addition, the audio hardware may includecircuitry for processing analog input signals to provide digital audiosignals for sharing with other devices.

The memory in control circuit 30 can include, for example, flash memoryand/or non-volatile random access memory (NVRAM). In someimplementations, instructions (e.g., software) are stored in aninformation carrier. The instructions, when executed by one or moreprocessing devices (e.g., the processor or microcontroller in controlcircuit 30), perform one or more processes, such as those describedelsewhere herein. The instructions can also be stored by one or morestorage devices, such as one or more (e.g. non-transitory) computer- ormachine-readable mediums (for example, the memory, or memory on theprocessor/microcontroller). As described herein, the control circuit 30(e.g., memory, or memory on the processor/microcontroller) can include acontrol system including instructions for controlling conversationassistance functions according to various particular implementations. Itis understood that portions of the control system (e.g., instructions)could also be stored in a remote location or in a distributed location,and could be fetched or otherwise obtained by the control circuit 30(e.g., via any communications protocol described herein) for execution.The instructions may include instructions for controlling conversationassistance functions, as well as digital signal processing andequalization. Additional details may be found in U.S. Patent ApplicationPublication 20140277644, U.S. Patent Application Publication20170098466, and U.S. Patent Application Publication 20140277639, thedisclosures of which are incorporated here by reference in theirentirety.

Personal audio device 10 can also include a sensor system 36 coupledwith control circuit 30 for detecting one or more conditions of theenvironment proximate personal audio device 10. Sensor system 36 caninclude one or more local sensors (e.g., inner microphones 18 and/orouter microphones 24) and/or remote or otherwise wireless (orhard-wired) sensors for detecting conditions of the environmentproximate personal audio device 10 as described herein. As describedfurther herein, sensor system 36 can include a plurality of distinctsensor types for detecting location-based conditions proximate thepersonal audio device 10.

According to various implementations, the conversation assistancedevices (which may be, for example, personal audio device 10 of FIG. 1)described herein can be personalized according to user feedback. Theseparticular implementations can allow a user to experience dynamic,personalized conversation assistance throughout differing acousticenvironments. These implementations can enhance the user experience incomparison to conventional conversation assistance systems.

As described with respect to FIG. 1, control circuit 30 can execute (andin some cases store) instructions for controlling audio functions inpersonal audio device 10 and/or a smart device coupled with the personalaudio device 10 (e.g., in a network). As shown in FIG. 2, controlcircuit 30 can include a conversation assistance audio engine 210configured to implement modifications in audio settings (e.g., settingsin ANR circuits 26A,B, FIG. 1) for outputs at the transducer (e.g.,speaker) 28 (FIG. 1) based upon user feedback to one or more prompts.Additionally, one or more portions of the conversation assistance audioengine 210 (e.g., software code and/or logic infrastructure) can bestored on or otherwise accessible to a smart device 280, which may beconnected with the control circuit 30 by any communications connectiondescribed herein. As described herein, particular functions of theconversation assistance audio engine 210 can be beneficially employed onthe smart device 280.

In particular, FIG. 2 shows a schematic data flow diagram illustrating acontrol process performed by conversation assistance audio engine 210 inconnection with a user 225. It is understood that in variousimplementations, user 225 can include a human user. FIG. 3 shows aprocess flow diagram illustrating processes performed by conversationassistance audio engine 210 according to various implementations. FIGS.4-6 show schematic depictions of an interfaces presented to the user 225to perform control operations of the personal audio device 10. FIGS. 1-6are referred to simultaneously, with particular emphasis on FIGS. 2 and3.

Returning to FIG. 2, data flows between conversation assistance audioengine 210 and other components in personal audio device 10 are shown.It is understood that one or more components shown in the data flowdiagram may be integrated in the same physical housing, e.g., in thehousing of personal audio device 10, or may reside in one or moreseparate physical locations.

Conversation assistance audio engine 210 can be coupled (e.g.,wirelessly and/or via hardwired connections in personal audio device 10)with an audio library 240, which can include simulated audio environmentfiles 250 for playback (e.g., streaming) at personal audio device 10.

Audio library 240 can be associated with digital audio sourcesaccessible via network interface 34 (FIG. 1) described herein, includinglocally stored, remotely stored or Internet-based audio libraries.Simulated audio environment files 250 are configured for playback at thepersonal audio device 10 to provide the user 225 with an immersive audioexperience similar to an environment that the user 225 may encounterwhile wearing the personal audio device 10. In some particularimplementations, a simulated audio environment file 250 can includeplayback of a speaker (e.g., a person speaking) along with backgroundnoise or other ambient sound. In certain cases, the speaker is playedback at a decibel level that is greater than the decibel level of thebackground noise. The simulated audio environment files 250 can includebinaural recordings of an environment, for example, a binaural recordingof a speaker in an automobile, or a binaural recording of a speaker atan airport lounge.

In certain implementations, a simulated audio environment can include acoffee shop or restaurant, train station, airplane cabin, live sportingvenue, etc. These simulated audio environments replicate real-worldenvironments that can make hearing challenging for many users. Inparticular cases, the simulated audio environment file 250 initiatesaudio playback at the personal audio device 10 at a signal-to-noise(SNR) ratio in which audibility limits intelligibility. In some cases,this SNR range is between approximately −5 dB SNR and approximately 20dB SNR. In certain implementations, a plurality of simulated audioenvironments are available for playback at a substantially identicalbackground noise level. In these implementations, a user 225 canexperience a set of two or more simulated audio environments at asubstantially identical background noise level in order to improve theaccuracy of the feedback received from that user 225 about that type ofenvironment. In one example, the user 225 is exposed to at least twosimulated audio environments within a narrow range of background noise(e.g., approximately 5 dB SNR or less of background noise variation) andasked for feedback about those environments. In a particular case, eachof the simulated audio environments can simulate a common setting, e.g.,a coffee shop or sporting venue, but with a distinct speaker in eachenvironment.

Conversation assistance audio engine 210 can also be coupled with asettings library 260 for controlling audio setting(s) on the personalaudio device 10. The settings library 260 can include a group of audiosettings 270 for applying different modifications to incoming acousticsignals received at the personal audio device 10. As described herein,the settings 270 can be adjusted based upon user feedback in response toexperiencing one or more of the simulated audio environments. In certaincases, adjusting the audio settings 270 in the personal audio device caninclude adjusting one or more of: a directivity of a microphone array inthe personal audio device 10, a microphone array filter on themicrophone array in the personal audio device 10, a volume of audioprovided to the user 225 at the personal audio device 10, parameterscontrolling wide dynamic range compression gain parameters controllingthe shape of the frequency versus gain function or per-ear settings(e.g., dynamic range compression parameters on a per-ear basis).

When the simulated audio environment file 250 is played back at thepersonal audio device 10, the user 225 hears the intended SNR for thatparticular simulated audio environment according to current settings 270for the personal audio device 10. That is, the simulated audioenvironment file 250 is configured to provide the user 225 with audioplayback at a specified sound pressure level, background/foregroundnoise ratio, and noise cancelling level to simulate how that environmentwould be perceived according to the current settings 270 on the personalaudio device 10. Additionally, the simulated audio environment file 250can include any effect of signal processing found in a hearing aid. Insome particular implementations, the simulated audio environment file250 can also include a video file (or be paired with a correspondingvideo file) for playback on an interface (e.g., an interface on a smartdevice 280 and/or on the personal audio device 10. The video file can besynchronized with the audio playback at the personal audio device 10 toprovide the user 225 with an immersive simulated experience. Playback ofthe audio and/or video can be controlled by the control circuit 30and/or a processing component on a paired smart device 280. In variousimplementations, the simulated audio environment file 250 can be storedon the smart device 280, streamed from another source (e.g., cloud-basedstorage device(s)) and/or stored on the personal audio device 10. Inparticular implementations where the personal audio device 10 does nothave wireless technology, the simulated audio environment file(s) 250can be stored on the personal audio device 10, enabling a stand-alonepersonalization process. In other particular implementations where thepersonal audio device 10 relies upon the Bluetooth Low Energy (BLE)communication protocol, high-fidelity audio streaming may not bepossible. In these cases, the simulated audio environment file 250 isbeneficially stored on the personal audio device 10 in order to enablehigh-fidelity playback that is not degraded by the low-bitrate encodingof the BLE link.

As noted herein, conversation assistance audio engine 210 can also becoupled with smart device 280 that has access to one or more userprofiles 290 (e.g., in a profile system 300) or biometric informationabout user 225. It is understood that smart device 280 can include oneor more personal computing devices (e.g., desktop or laptop computer),wearable smart devices (e.g., smart watch, smart glasses), a smartphone, a remote control device, a smart beacon device (e.g., smartBluetooth beacon system), a stationary speaker system, etc. Smart device280 can include a conventional user interface for permitting interactionwith user 225, and can include one or more network interfaces forinteracting with control circuit 30 and other components in personalaudio device 10 (FIG. 1). In some example implementations, smart device280 can be utilized for: connecting personal audio device 10 to a Wi-Finetwork; creating a system account for the user 225; setting up musicand/or location-based audio services; browsing of content for playback;setting preset assignments on the personal audio device 10 or otheraudio playback devices; transport control (e.g., play/pause, fastforward/rewind, etc.) for the personal audio device 10; and selectingone or more personal audio devices 10 for content playback (e.g., singleroom playback or synchronized multi-room playback). In some cases smartdevice 280 may also be used for: music services setup; browsing ofcontent; setting preset assignments on the audio playback devices;transport control of the audio playback devices; and selecting personalaudio devices 10 (or other playback devices) for content playback. Smartdevice 280 can further include embedded sensors for measuring biometricinformation about user 225, e.g., travel, sleep or exercise patterns;body temperature; heart rate; or pace of gait (e.g., viaaccelerometer(s)). As noted herein, smart device 280 can be used tosynchronize playback of simulated audio environment files 250 to providean immersive audio (and in some cases, video) simulation to the user225. Further, it is understood that one or more functions of theconversation assistance audio engine 210 can be stored, accessed and/orexecuted at smart device 280.

User profiles 290 may be user-specific, community-specific,device-specific, location-specific or otherwise associated with aparticular entity such as user 225. User profiles 290 can includeuser-defined playlists of digital music files, audio messages stored bythe user 225 or another user, or other audio files available fromnetwork audio sources coupled with network interface 34 (FIG. 1), suchas network-attached storage (NAS) devices, and/or a DLNA server, whichmay be accessible to the personal audio device 10 (FIG. 1) over a localarea network such as a wireless (e.g., Wi-Fi) or wired (e.g., Ethernet)home network, as well as Internet music services such as Pandora®,vTuner®, Spotify®, etc., which are accessible to the audio personalaudio device 10 over a wide area network such as the Internet. In somecases, profile system 300 is located in a local server or a cloud-basedserver, similar to any such server described herein. User profile 290may include information about audio settings associated with user 225 orother similar users (e.g., those with common hearing attributes ordemographic traits), frequency with which particular audio settings arechanged by user 225 or other similar users, etc. Profile system 300 canbe associated with any community of users, e.g., a social network,subscription-based music service, and may include audio preferences,histories, etc. for user 225 as well as a plurality of other users. Inparticular implementations, profile system 300 can include user-specificpreferences (as profiles 290) for audio settings 270. Profiles 290 canbe customized according to particular user preferences, or can be sharedby users with common attributes.

Conversation assistance audio engine 210 is also configured to receivesensor data from sensor system 36. Additionally, as noted herein, theconversation assistance audio engine 210 can receive sensor data fromthe smart device 280. This sensor data can be used to control variousfunctions such as ANR (and CNC) functions, dynamic volume control,notifications, etc. In some cases, sensor system 36 can include one ormore of the following sensors: a position tracking system; anaccelerometer/gyroscope; a microphone (e.g., including one or moremicrophones, which may include or work in concert with microphones 18and/or 24); and a wireless transceiver. These sensors are merelyexamples of sensor types that may be employed according to variousimplementations. It is further understood that sensor system 36 candeploy these sensors in distinct locations and distinct sub-componentsin order to detect particular environmental information relevant to user225 and the personal audio device 10.

A position tracking system can include one or more location-baseddetection systems such as a global positioning system (GPS) locationsystem, a Wi-Fi location system, an infra-red (IR) location system, aBluetooth beacon system, etc. In various additional implementations, theposition tracking system can include an orientation tracking system fortracking the orientation of the user 225 and/or the personal audiodevice 10. The orientation tracking system can include a head-trackingor body-tracking system (e.g., an optical-based tracking system,accelerometer, magnetometer, gyroscope or radar) for detecting adirection in which the user 225 is facing, as well as movement of theuser 225 and the personal audio device 10. The position tracking systemcan be configured to detect changes in the physical location of thepersonal audio device 10 and/or user 225 (where user 225 is separatedfrom personal audio device 10) and provide updated sensor data to theconversation assistance audio engine 210. The position tracking systemcan also be configured to detect the orientation of the user 225, e.g.,a direction of the user's head, or a change in the user's orientationsuch as a turning of the torso or an about-face movement. In particularimplementations, the orientation tracking system can be used torecommend settings by comparing real-world characteristics (e.g., headtracking position) of the user's orientation with simulation preferencesfor the user 225. In additional implementations, head-tracking can beused to adjust the simulations for a more realistic personalizationexperience. For example, the orientation tracking system can include ahead tracking engine for detecting the head position of user 225, andthe conversation assistance audio engine 210 can adjust playback of thesimulated audio environment based upon the head position of the user225. The conversation assistance audio engine 210 can use head-relatedtransfer functions (HRTFs) to simulate the audio environment relative toknown position of the user's head in space from the orientation trackingsystem

An accelerometer/gyroscope can include distinct accelerometer componentsand gyroscope components, or could be collectively housed in a singlesensor component, e.g., an inertial measurement unit (IMU). Thiscomponent may be used to sense gestures based on movement of the user'sbody (e.g., head, torso, limbs) while the user is wearing the personalaudio device 10 or interacting with another device (e.g., smart device280) connected with personal audio device 10. As with any sensor insensor system 36, the accelerometer/gyroscope may be housed withinpersonal audio device 10 or in another device connected to the personalaudio device 10.

The microphone (which can include one or more microphones, or amicrophone array) can have similar functionality as the microphone(s) 18and 24 shown and described with respect to FIG. 1, and may be housedwithin personal audio device 10 or in another device connected to thepersonal audio device 10. As noted herein, microphone(s) may include orotherwise utilize microphones 18 and 24 to perform functions describedherein. Microphone(s) can be positioned to receive ambient acousticsignals (e.g., acoustic signals proximate personal audio device 10). Insome cases, these ambient acoustic signals include speech/voice inputfrom user 225 to enable voice control functionality. In some otherexample implementations, the microphone can detect the voice of user 225and/or of other users proximate to or interacting with user 225. Inparticular implementations, conversation assistance audio engine 210 isconfigured to analyze one or more voice commands from user 225 (viamicrophone), and modify the applied audio settings 270 on the personalaudio device 10. In some cases, the conversation assistance audio engine210 can include sensor data processing logic for analyzing voicecommands, including, e.g., natural language processing (NLP) logic orother similar logic.

As noted herein, the sensor system 36 can also include a wirelesstransceiver (comprising a transmitter and a receiver), which mayinclude, a Bluetooth (BT) or Bluetooth Low Energy (BTLE) transceiver orother conventional transceiver device. The wireless transceiver can beconfigured to communicate with other transceiver devices in distinctcomponents (e.g., smart device 280).

It is understood that any number of additional sensors could beincorporated in sensor system 36, and could include temperature sensorsor humidity sensors for detecting changes in weather withinenvironments, optical/laser-based sensors and/or vision systems fortracking movement or speed, light sensors for detecting time of day,additional audio sensors (e.g., microphones) for detecting human orother user speech or ambient noise, etc.

According to various implementations, control circuit 30 includes theconversation assistance audio engine 210, or otherwise accesses programcode for executing processes performed by conversation assistance audioengine 210 (e.g., via network interface 34). Conversation assistanceaudio engine 210 can include logic for processing feedback from the user225 about the simulated audio environments. Additionally, conversationassistance audio engine 210 can include logic for looking up andadjusting audio settings according to feedback received from the user225 and known characteristics of the simulated audio environment. Theconversation assistance audio engine 210 can also include logic forprocessing sensor data from the sensor system, e.g., data about ambientacoustic signals from microphones, data about a location of the personalaudio device 10, biometric data from a smart device, and/or usage datafrom a smart device.

In various implementations, the conversation assistance audio engine 210presents the user 225 of the personal audio device (e.g., conversationassistance audio device) 10 with a set of simulated audio environments,which can also include video playback, e.g., at a video player such assmart device 280. The simulated audio environment files 250 can beplayed back at the transducers (e.g., transducers 28A,B, FIG. 1) at thepersonal audio device 10. The user 225 provides feedback about eachsimulated audio environment, either directly to the control circuit 30,e.g., via an interface on the personal audio device 10, or to smartdevice 280 (e.g., via an interface).

As noted herein, conversation assistance audio engine 210 can includelogic for performing audio control functions according to variousimplementations. FIG. 3 shows a flow diagram illustrating processes inconversation assistance audio control performed by conversationassistance audio engine 210 and its associated logic.

As also noted herein, the conversation assistance audio engine 210 isconfigured to present the user 225 of the personal audio device 10 witha set of simulated audio environments (simulated audio environment files250), played back at personal audio device 10 (process 410, FIG. 3). Ifmore than one simulated audio environment is presented to the user 225,those simulated audio environments can be presented in a series, withone following another after a designated period. As described herein, invarious implementations when two or more simulated audio environmentsare presented to the user 225, those environments can be presented witha substantially identical background noise level. In certainimplementations, a preliminary process (not shown) can include the user225 initiating a conversation assistance personalization (or testing)process, for example, via an interface on the smart device 280 and/orvia an interface on the personal audio device 10. In some cases, theuser 225 can initiate the personalization/testing process through asoftware application (or, app) running on the smart device 280 and/orthe personal audio device 10. In other cases, the conversationassistance audio engine 210 can prompt the user 225 to begin apersonalization process, e.g., using any prompt described herein.

According to various implementations, the set of simulated audioenvironments are presented to the user with a predefined set of audiosettings 270. In certain cases, these predefined settings 270 aredefault settings for the personal audio device 10, e.g., standardsettings designed to function most effectively for the population ofpotential users of personal audio device 10 and similar devices. Inother cases, the predefined settings are saved in the personal audiodevice 10 based upon prior usage, e.g., if the user 225 or another prioruser of the personal audio device 10 has already defined settings forthe device. In still other cases, the predefined settings are based uponone or more user profile(s) 290, which can be attributed to the user 225and/or to other users. In certain cases, the profile-based settings canbe defined by settings selected or positively verified by a plurality ofusers in a community or network.

After presenting the user 225 with the set of simulated audioenvironments, the conversation assistance audio engine 210 is configuredto receive feedback from the user 225 about the about each simulatedaudio environment (process 420, FIG. 3). The conversation assistanceaudio engine 210 can include feedback logic 310 for analyzing thefeedback from the user 225 about the simulated audio environment file250. In some cases, feedback includes negative feedback from user 225about the simulated audio environment file 250 (e.g., “I cannot hearthis speaker clearly”, “No”, “Change”, “Louder”, “Quieter” or a similarresponse to a user interface prompt such as a thumbs-down, “No” vote,etc.). In other cases, feedback includes positive feedback from user 225about the simulated audio environment file 250 (e.g., “Yes”, “Good”,“Loud and clear” or a similar response to a user interface prompt suchas a thumbs-up, “Yes” vote, etc.). In various implementations, user 225can provide either a verbal response or a response to a user interfaceprompt. In particular cases, the feedback can include adjustment of aninterface control such as a knob, button, toggle, or voting mechanism.

Example interfaces are shown in FIGS. 4-6, which illustrate just some ofthe various functions of the personal audio device 10 and theconversation assistance audio engine 210. For example, FIG. 4 shows anexample interface 500 for adjusting tone settings in the personal audiodevice. The interface 500 is shown at an initial state (a) and afollow-up state (b). This example interface 500 depicts a speaker in ashop environment such as a coffee shop. The interface 500 allows theuser 225 to provide feedback about the characteristics of the simulatedaudio environment file 250 while listening to that file. While wearingthe personal audio device 10, the user 225 can interact with theinterface 500 on a smart device 280 and/or the personal audio device 10.In this particular example, the interface 500 includes a plurality ofactuatable mechanisms (e.g., buttons) 510 for providing feedback to theconversation assistance audio engine 210. The user 225 can selectbetween tones A and B (shown in initial state (a)), and then providefeedback about those tones (shown in follow-up state (b)). In thisexample implementation, the user 225 can actuate one of the actuatablemechanisms 510 such as tones A or B (e.g., via a touch screen command,voice command, or gesture command), and can provide feedback about thetones using interface 500 or another interface such as a voice orgesture-based interface. In the particular example shown, the user 225can elect to use the touch screen to initiate tone A and tone B, andthen provide feedback in a weighted scoring mechanism 520 (which mayinclude buttons or similar actuatable mechanism(s)). In this example,the weighted scoring mechanism 520 provides a plurality of optionsbetween the two tones A and B, e.g., “A sounds much better”, “A soundsbetter”, “A and B sound the same”, “B sounds better” and “B sounds muchbetter”. In particular implementations, the interface 500 can includeinstructions, e.g., text instructions on an interface (as shown ininitial state (a)), or audio instructions provided via one or moretransducers. While a weighted scoring mechanism is shown in FIG. 4,additional actuatable mechanisms could also be used to solicit feedback,such as a volume adjustor and/or treble/bass adjustor.

FIG. 5 shows an additional example interface 600. In this example, theinterface 600 can be used to control the settings for playback volume.This example interface 600 depicts a speaker in a restaurant environmentsuch as an outdoor café. The interface 600 allows the user 225 toprovide feedback about the subjective quality of the simulated audioenvironment file 250 while listening to that file. While wearing thepersonal audio device 10, the user 225 can interact with the interface600 on the smart device 340 and/or the personal audio device 10. In thisparticular example, the interface 600 includes at least one actuatablemechanism (e.g., volume adjustor 610) for providing feedback to theconversation assistance audio engine 210 about the volume of theplayback of the simulated audio environment file 250. The user 225 canslide the adjustor 610 across a range of values to modify the playbackvolume, and then actuate another actuatable mechanism (e.g., button 620)to verify that the volume level is acceptable (e.g., “That SoundsGood”). The user 225 can also adjust the volume adjustor and provide avoice command or gesture command to verify that the volume level isacceptable. In particular implementations, the interface 600 can includeinstructions, e.g., text instructions on an interface (as shown ininitial state (a)), or audio instructions provided via one or moretransducers.

FIG. 6 shows three distinct interfaces 700, 800, 900, each for adjustingvolume settings in the personal audio device 10 via the conversationassistance audio engine 210. These interfaces 700, 800, 900 are eachassociated with a distinct simulated audio environment file 250, thatis, each of the interfaces can be presented to the user 225 at adistinct time (e.g., in series, or separated by a designated period). Inthis example implementation, interfaces 700, 800, 900 can depictdifferent speakers (e.g., employees) at a restaurant. Similar tointerface 600 in FIG. 5, each of these interfaces 700 can include atleast one actuatable mechanism (e.g., volume adjustor 610) for providingfeedback to the conversation assistance audio engine 210 about thevolume of the playback of the simulated audio environment file 250. Theuser 225 can slide the adjustor 610 across a range of values to modifythe playback volume, and then actuate another actuatable mechanism(e.g., button 620) to verify that the volume level is acceptable (e.g.,“That Sounds Good”). The user 225 can also adjust the volume adjustorand provide a voice command or gesture command to verify that the volumelevel is acceptable. In the example depiction in FIG. 6, the simulatedaudio environment file 250 associated with each of the interfaces 700,800, 900 can include binaural playback at a substantially identicalsound pressure level, with background and speaker volume ratiosremaining approximately equal across all samples. In this scenario, theuser 225 is provided with three visually distinct simulations that havesubstantially identical audio characteristics. As discussed furtherherein, using two or more simulated audio environments can aid theconversation assistance audio engine 210 in selecting a best-fit audiosetting(s) (or a plurality of situation-based best-fit audio settings)for the personal audio device 10, for example, using feedback from allof the simulated audio environments scored by the user 225.

While some examples of feedback are described with respect to a userinterface, it is understood that feedback and/or other audio inputs suchas sensor data can include an audio signal, and may be analyzed usingacoustic feature extraction for one or more features including: energy,zero-crossing rate, mel-frequency cepstral coefficients, spectralflatness, summary statistics (e.g., mean, variance, skew or kurtosis) onany signal measurement, tempo/beats-per-minute and acousticfingerprinting. In some cases, conversation assistance audio engine 210(and logic therein) can use these acoustic features from feedback orother audio inputs, and metadata features from the simulated audioenvironment files 250, to perform statistical and probabilistic modelingin order to recommend or select audio settings 270 for the user 225 inaccordance with other functions. As noted herein, in variousimplementations, the statistical and probabilistic modeling enables theconversation assistance audio engine 210 to recommend or select audiosettings 270 for real-world usage (outside of the simulated environment)based upon the user adjustments in the simulated environment.

Returning to FIG. 2, and as noted herein, the conversation assistanceaudio engine 210 can include feedback logic 310 for processing feedbackreceived from the user 225, e.g., via one or more interfaces. Thefeedback logic 310 can be connected with settings lookup/adjustmentlogic 320, as well as sensor data processing logic 330, and can providefeedback, e.g., to the settings lookup/adjustment logic 320 for use inadjusting the settings 270 on the personal audio device 10. The sensordata processing logic 330 can also provide sensor data to the settingslookup/adjustment logic 320 for use in adjusting the settings 270 on thepersonal audio device 10. In some cases, feedback logic 310 can beconfigured to teach sensor data processing logic 330 and settingslookup/adjustment logic 320 about preferences of user 225, e.g., whereone or more of these logic components includes an artificialintelligence (AI) component for iteratively refining logic operations toenhance the accuracy of its results. Example AI components could includemachine learning logic, a neural network including an artificial neuralnetwork, a natural language processing engine, a deep learning engine,etc. In any case, feedback logic 310 can be configured to analyzefeedback and enhance future operations of conversation assistance audioengine 210. It is further understood that feedback logic 310, settingslookup/adjustment logic 320 and/or sensor data processing logic 330 maybe interconnected in such a manner that these components act in concertor in reliance upon one another.

In some cases, the feedback logic 310 is configured to process a commandor request from the user 225 and categorize that command or request interms of a settings adjustment for the personal audio device 10. Forexample, the feedback logic 310 can be configured to translate the usercommands and/or requests made via one or more interfaces into a valueadjustment for the corresponding audio settings 270. As noted herein,this value adjustment is sent to the settings lookup/adjustment logic320 for determining a corresponding settings adjustment based upon theuser feedback. The feedback logic 310 can be configured to translate theadjustment within the simulated audio environment file 250 into anadjustment value for one or more settings on the personal audio device10. That is, the adjustment within the simulation interface may involvea manageable number of variables for the user 225, such as tone orvolume, but the audio settings 270 can include a variety of potentialadjustments to meet the desired audio output of the user 225 at thepersonal audio device 10. In this sense, the feedback logic 310translates the adjustment to the variables in the simulation withadjustment values to audio settings on the personal audio device 10,e.g., a directivity of a microphone array, a microphone array filter, avolume of audio provided to the user, parameters controlling widedynamic range compression or gain parameters controlling the shape ofthe frequency versus gain function. In some cases, these translationsare made using a relational database or indexed data structure.

In some example implementations, the user 225 can set a desired widedynamic range compression (WDRC) value (also referred to as World Volumeherein) on the personal audio device 10 using interface adjustmentsbased upon the simulated audio environment file 250. Additional exampleimplementations can include simulating multiple sound pressure levels(SPLs) in the simulated audio environment file 250, and adjusting theWDRC (in the audio settings 270) to match a real-world environment basedupon one or more selected SPLs from the simulated environment. While SPLand WDRC are used as examples, the conversation assistance audio engine210 can be configured to actively make setting adjustments, or recommendsettings (or adjustments) based upon any variable correlated between thesimulated environment and a real-world environment.

As shown in FIG. 3, after receiving feedback from the user 225 about thesimulated audio environment(s), the conversation assistance audio engine210 is further configured to adjust at least one audio setting at thepersonal audio device 10 based upon the feedback from the user 225 andknown audio characteristics of the set of simulated audio environments(simulated audio environment files 250) and the personal audio device 10(process 430, FIG. 3).

In some particular implementations, this process can include selecting abest-fit audio setting for the conversation assistance audio devicebased upon the feedback from the user 225. In these cases, the settingslookup/adjustment logic 320 can select a closest setting value for theadjustment indicated by the feedback (as provided by feedback logic310). In some cases, the best-fit audio setting can be selected usingfeedback from the user 225 about two or more simulated audioenvironments, e.g., where the user 225 provides distinct feedback in atleast one adjustment factor across one or more of the audioenvironments. In this case, the conversation assistance audio engine 210(e.g., feedback logic 310) can average, weight or otherwise account fordifferences in feedback values for adjustment mechanisms across aplurality of simulated audio environments. In some particularimplementations, the conversation assistance audio engine 210 candetermine a best-fit audio setting adaptively, e.g., by requiring thatthe user 225 select a setting in a sufficient number of simulated audioenvironments or with a narrow enough range between value selections tomeet a threshold confidence interval. In other cases, the conversationassistance audio engine 210 can select the best-fit setting oncefeedback is received across a threshold number of settings at distinctsound pressure levels. The conversation assistance audio engine caninterpolate a WDRC setting value for the personal audio device using aWDRC v. input SPL curve. Additionally, the feedback logic 310 and/orsettings lookup/adjustment logic 320 can include an error metric thatcorrelates the difference between the recommended audio setting 270 fromthe user adjustments in the simulation and the audio setting 270 thatthe user 225 ultimately selects after use in a real-world setting. Theerror metric can be updated over time to train the logic and minimizethe difference between the recommended and selected settings.

In some particular implementations, the audio setting(s) 270 adjustmentis scaled based upon a scaling factor. For example, a scaling factor canbe applied to the audio setting adjustment to account for distinctionsbetween user perception of audio content in a demonstration settingversus in a real-world setting. In one particular implementation, thescaling factor includes an inference between audio characteristics ofeach of the simulated audio environments and a model of real-world audiocharacteristics. This inference can be based upon data gathered fromtest users, published research and/or user feedback. For example, thescaling factor can be saved in a user profile 290 or can be applied bysettings lookup/adjustment logic 320 according to user settings, profilecharacteristics, feedback received from the user 225 or data gatheredfrom the sensor system 36. In one example, the conversation assistanceaudio engine 210 applies a positive scaling factor to user adjustments,e.g., to volume adjustments, including positively scaling the audiosetting 270 on the personal audio device 10. This positive scalingfactor infers that users will adjust the setting (e.g., volume) in thesimulated environment to a lesser extent than they would adjust thatsame setting in a real-world use environment. In variousimplementations, the scaling factor(s) can be adjusted over time, usingany machine learning engine described herein. For example, individualscaling factors can be refined over time based upon adjustments made bythe user 225 to his/her personal audio device 10. Additionally, defaultscaling factors for the personal audio device 10 can be adjusted and/orrefined based upon adjustment data gathered from personalizationadjustments and/or demonstration-based adjustments made by users of agroup of audio devices. As a greater number of users performadjustments, the conversation assistance audio engine 210 can identifytrends in usage and improve scaling factor accuracy using one or moreinput parameters, e.g., age, gender, initial World Volume setting, etc.

In some particular implementations, the conversation assistance audioengine 210 is further configured to create a set of distinct audio modesbased upon the feedback from the user 225 and known audiocharacteristics of the set of simulated audio environments and thepersonal audio device 10. For example, the distinct audio modes can becreated according to user responses from one or more simulated audioenvironments. In certain cases, the conversation assistance audio engine210 can create a set of user-selectable audio modes each correlated withdistinct audio filters configured to enhance audibility of a range offrequencies for specific acoustic environments. According to someimplementations, the distinct audio modes can be intended for use indistinct real-world environments. Audio modes can include “quietconversation”, “loud room”, “low-frequency noise”, or similarcategories. The conversation assistance audio engine 210 can dynamicallyswitch between modes based upon user feedback and/or sensor data fromthe sensor system 36. In some cases, the conversation assistance audioengine 210 can switch between distinct audio modes in response todetecting a change in an acoustic signature of ambient sound detected atthe conversation assistance audio device. In particular examples, theconversation assistance audio engine 210 can switch between audio modeswhen a sound pressure level of ambient noise passes a threshold level(either up or down), when a sensor such as a microphone detects that theuser 225 is raising his/her voice about an established average decibelrange, and/or a sensor (e.g., microphone) detects an acoustic signaturesuch as the low frequency sound of an airplane cabin during flight.

In some implementations, the conversation assistance audio engine 210can provide a suggested audio mode for the personal audio device 10 tothe user 225 based upon the feedback from the user 225 and the knownaudio characteristics of the simulated audio environment(s) and thepersonal audio device 10, prior to adjusting the setting(s) 270 on thepersonal audio device 10. In this example, the conversation assistanceaudio engine 210 can present the user 225 with a suggested audio modesetting prior to making any adjustment to the setting(s) 270. Withreference to the flow diagram in FIG. 3, this optional process is shownas process 425, which can be performed between process 420 and process430, providing the user 225 with a potential audio mode for use of thepersonal audio device 10 after processing that user's feedback. Theconversation assistance audio engine 210 can notify the user 225 aboutthe suggested audio mode via any mechanism described herein, e.g., viaan interface prompt such as a notification on the smart device 280 orthe personal audio device 10 (“consider Quiet Conversation setting”)and/or a voice prompt via the transducers 28 on the personal audiodevice 10.

As noted herein, the conversation assistance audio engine 210 can beconfigured to adjust at least one of the audio settings 270 based uponuser feedback about the simulated audio environment(s) and knowncharacteristics of the environment(s) and the personal audio device 10.In some cases, the audio settings 270 can be adjusted by applying asignal processing modification in the personal audio device 10 or acomputing device (e.g., smart device 280) connected with the personalaudio device 10. The signal processing modifications can include anyadjustments described herein, e.g., wide dynamic range compressionparameters, directivity, etc. In certain cases, the signal processingmodification is performed as a weighted adjustment, such that a firstadjustment of one parameter does not have a significant impact on theaudio settings 270 (e.g., where the adjustment was made in error). Inthese cases, subsequent similar adjustments can have a greater impact onthe audio settings 270. Weighted adjustment can be used to limitunintended impacts on the audio settings 270, and can be performed usingtechniques such as N-point running-average, median filtering, etc.

FIG. 7 shows a flow diagram illustrating an additional approach forpersonalizing a conversation assistance audio device (e.g., personalaudio device 10) according to various implementations. The processesillustrated can be performed by the conversation assistance audio engine210, and as such, FIG. 7 is referred to concurrently with FIGS. 1 and 2,with particular focus on FIG. 2.

In certain case, this process can include receiving a command from theuser 225 for assistance with audio settings 270 in personal audio device10 (process 710, FIG. 7). In various particular implementations, theuser command is a single interface command, such as a single tactilecommand (e.g., detectable by sensor system 36), a single gesture command(e.g., detectable by sensor system 36), or a single voice command (e.g.,detectable by microphones in personal audio device 10 and/or smartdevice 280). According to some implementations, the user 225 can easilyinitiate this personalization process by using a single interfacecommand, that is, a command that initiates a subsequent personalizationprocess without the need for an intervening command.

In response to receiving the single interface command from the user 225,the conversation assistance audio engine 210 (including feedback logic310, settings lookup/adjustment logic 320 and sensor data processinglogic 330) can apply a baseline set of audio settings to the personalaudio device 10 (process 720, FIG. 7). In certain cases the baseline setof audio settings are median audio settings for a respective pluralityof audio parameters (e.g., volume, tone, balance) or user-defined audiosettings for the respective plurality of audio parameters from a groupof users. Some instances may rely upon a baseline, or median, group ofaudio settings for the personal audio device 10 constructed from userprofiles 290. This baseline can be built from an average of a populationof users, such as all users of the personal audio device 10, or from aselected subset in that population (e.g., users that have used thedevice for a certain period, are of a certain age, or are of a certaingender). In other cases, the personal audio device 10 is configured torevert to default audio settings in response to receiving singleinterface command via the conversation assistance audio engine 210.

As noted herein, the baseline audio settings on the personal audiodevice 10 can be applied in response to the single interface commandwithout any intervening command or communication between the user 225and the personal audio device 10. In some examples, the user 225 taps aninterface button on the personal audio device 10, gestures using his/herbody (detected by sensor system 36), or makes a simple voice command(e.g., “Help”), and the conversation assistance audio engine 210 appliesthe baseline settings to the personal audio device 10 (without anintervening prompt or command from user 225).

In certain cases, when the baseline audio settings are applied to thepersonal audio device 10, the user 225 may wish to change thosesettings. In these cases, the conversation assistance audio engine 210can receive a user command to change the baseline audio settings(process 730, FIG. 7), e.g., via any interface described herein. Asdescribed with respect to process 710, the user command can be a singleinterface command, such as a single tactile command, a single gesturecommand, or a single voice command.

In response to receiving that user command, the conversation assistanceaudio engine 210 can apply a modified set of audio settings to thepersonal audio device 10 (process 740, FIG. 7). These modified audiosettings can be applied based upon usage of the personal audio device 10(e.g., from the user's profile 290 stored in profile system 300) and apopulation model of usage of similar personal audio devices 10 (e.g.,constructed from user profiles 290 stored in profile system 300). Incertain implementations, the conversation assistance audio engine 210presents the user 225 with the most common (e.g., the N-most common)adjustments made after users view the same or different simulated audioenvironment(s). In particular cases, the conversation assistance audioengine 210 presents the user 225 with these N-most common adjustmentssequentially. In other cases, the conversation assistance audio engine210 can apply a settings modification rule such as: increase or decreasea wide dynamic range compression parameter (or, “World Volume”) by anoticeable step size (e.g., 5 decibels).

According to various implementations, the conversation assistance audioengine 210 can be configured to iteratively adjust the audio settings270 on the personal audio device 10 in response to receiving usercommands, e.g., single interface commands. For example, as shown in FIG.7 (in loop 750), the conversation assistance audio engine 210 can beconfigured to iteratively receive a subsequent user command to changethe modified set of audio settings (looping back to process 730), andapply a further modified set of audio settings to the personal audiodevice 10 in response to receiving that subsequent user command (process740). As with the initial modification of audio settings, the subsequentmodification(s) can be applied based upon usage of the conversationassistance audio device by the user 225, as well as the population modelof usage of similar conversation assistance audio devices.

The conversation assistance audio engine 210 is described in someexamples as including logic for performing one or more functions. Invarious implementations, the logic in conversation assistance audioengine 210 can be continually updated based upon data received from theuser 225 (e.g., user selections or commands), sensor data received fromthe sensor system 36, settings updates (in settings library 260),updates and/or additions to the audio library 240 and/or updates to userprofile(s) 290 in the profile system 300.

In some example implementations, conversation assistance audio engine210 (e.g., using feedback logic 310, settings lookup/adjustment logic320 and/or sensor data processing logic 330) is configured to performone or more of the following logic processes using sensor data, commanddata and/or other data accessible via profile system 300, smart device280, etc.: speech recognition, speaker identification, speakerverification, word spotting (e.g., wake word detection), speech endpointing (e.g., end of speech detection), speech segmentation (e.g.,sentence boundary detection or other types of phrase segmentation),speaker diarization, affective emotion classification on voice, acousticevent detection, two-dimensional (2D) or three-dimensional (3D) beamforming, source proximity/location, volume level readings, acousticsaliency maps, ambient noise level data collection, signal qualityself-check, gender identification (ID), age ID, echocancellation/barge-in/ducking, language identification, and/or otherenvironmental classification such as environment type (e.g., small room,large room, crowded street, etc.; and quiet or loud).

In some implementations, conversation assistance audio engine 210 isconfigured to work in concert with sensor system 36 to continuallymonitor changes in one or more environmental conditions. In some cases,sensor system 36 may be set in an active mode, such as where a positiontracking system pings nearby Wi-Fi networks to triangulate location ofthe personal audio device 10, or a microphone (e.g., microphones 18and/or 24) remains in a “listen” mode for particular ambient sounds. Inother implementations, sensor system 36 and conversation assistanceaudio engine 210 can be configured in a passive mode, such as where awireless transceiver detects signals transmitted from nearby transceiverdevices or network devices. In still other implementations, distinctsensors in the sensor system 36 can be set in distinct modes fordetecting changes in environmental conditions and transmitting updatedsensor data to conversation assistance audio engine 210. For example,some sensors in sensor system 36 can remain in an active mode whileaudio device 10 is active (e.g., powered on), while other sensors mayremain in a passive mode for triggering by an event.

As described herein, user prompts can include an audio prompt providedat the audio device 10, and/or a visual prompt or tactile/haptic promptprovided at the audio device 10 or a distinct device (e.g., smart device280). In some cases, an audio prompt can include a phrase such as,“Would you like to personalize your conversation assistance device?,” or“Respond with a nod or “yes” to adjust audio settings,” or, “Take actionX to initiate personalization mode.” These are merely examples of audioprompts, and any suitable audio prompt could be used to elicit actuationby the user 225. In other cases, a visual prompt can be provided, e.g.,on a smart device 280 or at the audio device 10 (e.g., at a userinterface) which indicates that one or more operating modes ormodifications are available. The visual prompt could include anactuatable button, a text message, a symbol, highlighting/lowlighting,or any other visual indicator capable of display on the audio device 10and/or the smart device 280. A tactile/haptic prompt can include, e.g.,a vibration or change in texture or surface roughness, and can bepresented at the audio device 10 and/or smart device 280. Thistactile/haptic prompt could be specific to the conversation assistanceaudio engine 210, such that the tactile/haptic prompt is a signaturewhich indicates the operating mode (e.g., personalization mode) oradjustment (e.g., single-command adjustment) is available. As thetactile/haptic prompt may provide less information about the underlyingcontent offered, distinct tactile/haptic prompts could be used toreflect priority, e.g., based upon user profile(s) 290 or othersettings.

In some particular implementations, actuation of the prompt can bedetectable by the audio device 10, and can include a gesture, tactileactuation and/or voice actuation by user 225. For example, user 225 caninitiate a head nod or shake to indicate a “yes” or “no” response to aprompt, which is detected using a head tracker in the sensor system 36.In additional implementations, the user 225 can tap a specific surface(e.g., a capacitive touch interface) on the audio device 10 to actuatethe prompt, or can tap or otherwise contact any surface of the audiodevice 10 to initiate a tactile actuation (e.g., via detectablevibration or movement at sensor system 36). In still otherimplementations, user 225 can speak into a microphone at audio device 10to actuate the prompt and initiate the personalization functionsdescribed herein.

In some other implementations, actuation of the prompt is detectable bythe smart device 280, such as by a touch screen, vibrations sensor,microphone or other sensor on the smart device 280. In certain cases,the prompt can be actuated on the audio device 10 and/or the smartdevice 280, regardless of the source of the prompt. In otherimplementations, the prompt is only actuatable on the device from whichit is presented. Actuation on the smart device 280 can be performed in asimilar manner as described with respect to audio device 10, or can beperformed in a manner specific to the smart device 280.

The personalization processes described according to variousimplementations can significantly improve the user experience whencompared with conventional approaches, for example, by closely tailoringthe audio settings on the personal audio device 10 to user preferences.The personalization processes described according to variousimplementations can ease user setup of the personal audio device 10 andimprove conversation assistance functions during use. Additionally,certain implementations allow the user to change audio settings with asingle interface command, streamlining the process of adjustingsettings. Because the conversation assistance audio device (personalaudio device 10) is often used in a public setting, users can appreciatethe discreet nature of single interface command-based adjustments to theaudio settings.

The functionality described herein, or portions thereof, and its variousmodifications (hereinafter “the functions”) can be implemented, at leastin part, via a computer program product, e.g., a computer programtangibly embodied in an information carrier, such as one or morenon-transitory machine-readable media, for execution by, or to controlthe operation of, one or more data processing apparatus, e.g., aprogrammable processor, a computer, multiple computers, and/orprogrammable logic components.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program can be deployed to be executed on one computer or onmultiple computers at one site or distributed across multiple sites andinterconnected by a network.

Actions associated with implementing all or part of the functions can beperformed by one or more programmable processors executing one or morecomputer programs to perform the functions of the calibration process.All or part of the functions can be implemented as, special purposelogic circuitry, e.g., an FPGA and/or an ASIC (application-specificintegrated circuit). Processors suitable for the execution of a computerprogram include, by way of example, both general and special purposemicroprocessors, and any one or more processors of any kind of digitalcomputer. Generally, a processor will receive instructions and data froma read-only memory or a random access memory or both. Components of acomputer include a processor for executing instructions and one or morememory devices for storing instructions and data.

In various implementations, components described as being “coupled” toone another can be joined along one or more interfaces. In someimplementations, these interfaces can include junctions between distinctcomponents, and in other cases, these interfaces can include a solidlyand/or integrally formed interconnection. That is, in some cases,components that are “coupled” to one another can be simultaneouslyformed to define a single continuous member. However, in otherimplementations, these coupled components can be formed as separatemembers and be subsequently joined through known processes (e.g.,soldering, fastening, ultrasonic welding, bonding). In variousimplementations, electronic components described as being “coupled” canbe linked via conventional hard-wired and/or wireless means such thatthese electronic components can communicate data with one another.Additionally, sub-components within a given component can be consideredto be linked via conventional pathways, which may not necessarily beillustrated.

A number of implementations have been described. Nevertheless, it willbe understood that additional modifications may be made withoutdeparting from the scope of the inventive concepts described herein,and, accordingly, other embodiments are within the scope of thefollowing claims.

We claim:
 1. A computer-implemented method of personalizing aconversation assistance audio device, the method comprising: receiving acommand from a user for assistance with audio settings in theconversation assistance audio device; applying a baseline set of audiosettings to the conversation assistance audio device in response toreceiving the user command for assistance; receiving a user command tochange the baseline set of audio settings; and applying a modified setof audio settings to the conversation assistance audio device inresponse to receiving the user command to change the baseline set ofaudio settings, wherein the modified set of audio settings are appliedbased upon usage of the conversation assistance audio device and apopulation model of usage of similar conversation assistance audiodevices.
 2. The computer-implemented method of claim 1, wherein the usercommand comprises a single interface command.
 3. Thecomputer-implemented method of claim 2, wherein the single interfacecommand comprises a single voice command, a single tactile command, or asingle gesture command.
 4. The computer-implemented method of claim 2,wherein the baseline audio settings are applied in response to thesingle interface command without any intervening command.
 5. Thecomputer-implemented method of claim 2, further comprising iterativelyperforming the following: receiving a subsequent user command to changethe modified set of audio settings; and applying a further modified setof audio settings to the conversation assistance audio device inresponse to receiving the subsequent user command, wherein the furthermodified set of audio settings are applied based upon the usage of theconversation assistance audio device and the population model of usageof similar conversation assistance audio device, and wherein thesubsequent user command comprises a single interface command.
 6. Thecomputer-implemented method of claim 1, wherein the baseline set ofaudio settings comprise median audio settings for a respective pluralityof audio parameters.
 7. The computer-implemented method of claim 1,wherein the baseline set of audio settings comprise user-defined audiosettings for the respective plurality of audio parameters from a groupof users.
 8. The computer-implemented method of claim 1, wherein theaudio settings on the conversation assistance audio device comprise adirectivity of a microphone array in the conversation assistance audiodevice.
 9. The computer-implemented method of claim 1, wherein the audiosettings on the conversation assistance audio device comprise amicrophone array filter configuration on the microphone array in theconversation assistance audio device
 10. The computer-implemented methodof claim 1, wherein the audio settings on the conversation assistanceaudio device comprise a volume of audio provided to the user at theconversation assistance audio device.
 11. The computer-implementedmethod of claim 1, wherein the audio settings on the conversationassistance audio device comprise parameters controlling wide dynamicrange compression or gain parameters controlling the shape of thefrequency versus gain function.